feat(vllm-cpp): wire the full engine config surface through engine_args - #11159
Open
localai-bot wants to merge 9 commits into
Open
feat(vllm-cpp): wire the full engine config surface through engine_args#11159localai-bot wants to merge 9 commits into
localai-bot wants to merge 9 commits into
Conversation
The vllm-cpp backend could configure four of the engine's knobs - block size,
KV block count, max sequence length and max concurrent sequences - out of a
config surface that is considerably larger. Speculative decoding, prefix
caching, the chunked-prefill token budget, the scheduling policy and the
external KV connector were all reachable from vllm.cpp's own HTTP server and
from nothing LocalAI could write in a model config.
Part of that gap was the C ABI itself, which carried strictly less than
EngineParams does; that is fixed upstream in vllm.cpp ABI v9 (this bumps the pin
to it). The rest was here: the backend parsed a flat `options:` list with five
recognised keys and had no way to express a nested JSON document at all.
Configuration now goes through `engine_args:`, the same map the vLLM and SGLang
backends already take, with keys spelled as vLLM's own CLI flags - so a
`speculative_config` or `kv_transfer_config` block written for vLLM works
verbatim:
engine_args:
max_num_batched_tokens: 8192
enable_prefix_caching: true
scheduling_policy: lpm
speculative_config:
method: dflash
model: z-lab/Qwen3.6-27B-DFlash
num_speculative_tokens: 4
kv_transfer_config:
kv_connector: LMCacheConnector
kv_role: kv_both
kv_connector_extra_config: {host: 127.0.0.1, port: 65432}
The `options:` list keeps working, and now reads every key too, so no existing
config breaks; engine_args wins where both set the same key.
Two details worth calling out. `enable_prefix_caching: false` maps to the ABI's
force-OFF state (2), not the 0 that means "let the model capability decide" -
collapsing them would silently turn the cache ON for the dense architectures
that default it on. And cSamplingParams grows the ABI v8 logits-processor tail:
LocalAI installs no processor, but the C side reads those fields off the pointer
we hand it, so a Go struct that stopped short would have had the engine read 16
bytes past our allocation and call whatever sat there.
The importer gets the safetensors counterpart of the llama-cpp MTP hook: a
`vllm-cpp` import of a HuggingFace repo probes config.json and, on a checkpoint
that declares an MTP head, writes speculative_config {method: mtp} into the
generated engine_args. DFlash draft repositories are detected and refused with a
warning rather than configured as standalone models, since a drafter cannot
serve alone. The llama-cpp importer stops applying its own `spec_type:draft-mtp`
options when the chosen backend is vllm-cpp: those are llama.cpp option keys
vllm-cpp does not read, and vllm.cpp rejects MTP over a GGUF source anyway
because the `mtp.*` draft tensors only exist in the safetensors checkpoint.
docs/content/features/text-generation.md gains a vllm.cpp section - the backend
had no documentation page at all - covering the engine_args table, all three
speculative methods, LMCache, and the legacy options list.
Signed-off-by: Ettore Di Giacinto <mudler@localai.io>
Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
The Go PODs in govllmcpp.go are hand-written against one VLLM_ABI_VERSION and the Makefile pins the vllm.cpp commit that produces it. Nothing checked those two agree short of the e2e suite, which needs a model to run at all, so a pin bump could land with a stale mirror and only fail at a user's first load. VLLM_CPP_LIBRARY now drives a handshake spec that dlopens a built libvllm, binds every symbol, and compares the library's reported ABI against the mirrors'. No weights required. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
974d9d72 was the pre-merge commit on the feature branch. eec09bed is the merged main tip carrying the same ABI v9 surface, and is the tree the capi suite (31/31) and the Go ABI handshake were actually verified against. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…F cache The engine resolves speculative_config.model against a directory containing config.json, or against ~/.cache/huggingface/hub/models--<org>--<repo>/ snapshots/*, and it never downloads. LocalAI keeps models in its own directory, so the repo-id spelling the vLLM docs teach - "z-lab/Qwen3.6-27B-DFlash" - misses the HF cache and dies deep inside the load with "draft checkpoint not found", which reads like a broken checkpoint rather than a model nobody fetched. Resolve it before the load call: the reference as given, then its last path segment under LocalAI's models dir (what LocalAI's own downloader produces), then the whole reference under the models dir. When none resolve, fail there naming both what was asked for and every location tried, so the message says what to do about it. mtp and ngram pass through untouched - neither has a separate draft checkpoint. A speculative_config that does not parse also passes through, because the engine owns config validation and produces the better error. Docs also gain the two limits that were missing and are easy to lose an afternoon to: speculation is Qwen3.5/3.6-only at this engine pin regardless of format, and mtp/dflash need a safetensors target. The latter is a gap in the engine's GGUF loader rather than a property of GGUF - the format carries MTP weights fine, llama.cpp reads them as nextn.* tensors plus a <arch>.nextn_predict_layers key - so the docs say that rather than implying GGUF cannot express it. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
…ding vllm.cpp landed ABI v10 on main while this branch was open. The backend's runtime handshake refuses any library whose reported ABI differs from the mirrors', so the pin and the Go PODs move together or not at all. v10 appends one int32, `enable_jump_forward`, AFTER the v9 fields. Nothing else in the config surface changed: the SGLang reconciliation that carried it explicitly dropped its own duplicate scheduler_policy int in favour of the v9 `scheduling_policy` string this branch already wires, and a diff of EngineParams and the server flags across the window turns up jump forward and nothing else. So the exposure is one new knob, `engine_args.enable_jump_forward` - SGLang's grammar-speed subset, which emits grammar-forced tokens without spending a model step and therefore only affects constrained requests. It is the SECOND tri-state on this struct, and it repeats the trap the first one had: 0 is not "off", it is "defer" (to the environment here, to the model capability for prefix caching), so an explicit `false` has to reach the engine as 2. The bool->tri-state helper and the log renderer are now shared rather than duplicated per field, and named for the encoding instead of for prefix caching, since the next tri-state will want them too. The docs say this outright, because "omitting the key" and "setting it false" being different is not guessable. The Go mirror grows the field plus an EXPLICIT trailing pad: the struct is 8-aligned and now ends on a lone int32, so it is 88 bytes rather than 84. The offset assertions cover it, and the real-library handshake spec (VLLM_CPP_LIBRARY against a CPU libvllm.so built at the new pin) confirms the version agrees: 43 specs pass, ABI reported 10. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
localai-org-maint-bot
force-pushed
the
feat/vllm-cpp-engine-args
branch
from
July 29, 2026 09:07
f90497a to
df5e2a6
Compare
…ding is real now The pin sat at f384edcd while vllm.cpp main moved a long way. The ABI is unchanged at v10 and both POD structs are field-identical to the pinned commit (verified by diffing vllm_model_params and vllm_sampling_params across the range), so the Go mirror needs no edit and this is a clean bump. What it picks up matters for this backend: - MTP speculative decoding from a GGUF target, gated end to end on GPU. - DFlash speculative decoding with a GGUF draft AND a GGUF target. - NVFP4 GGUF: dequant, plus a native fp4 compute path for dense and full-attention projections. On the 27B that closed a cross-container divergence entirely (the GGUF and safetensors builds of the same quantization run now emit identical tokens) and halved peak RSS. - A real engine fix: the GDN speculative state gather/scatter was mis-striding the widened conv row, so speculation silently corrupted the target's own recurrent state on CPU. Docs corrected accordingly. The section previously told users that mtp and dflash are rejected on a .gguf target and called it a gap in the engine's GGUF loader. That is no longer true, and leaving it would send people to safetensors for no reason. A head-less GGUF is still refused, and the text now says so with the actual cause. The real-library ABI handshake was re-run against a libvllm.so built at the exact pinned commit rather than a stale one: 43 specs pass, reported ABI 10. make lint clean. Signed-off-by: Ettore Di Giacinto <mudler@localai.io> Assisted-by: Claude Code:claude-opus-5 [ClaudeCode]
Keep the GNU constant-folding diagnostics visible on Darwin without allowing vllm.cpp's global -Werror to fail the Metal backend build. Assisted-by: Codex:gpt-5 [systematic-debugging]
The previous no-error flag is overridden by vllm.cpp's later target-local -Werror. Disable only the Apple Clang folding diagnostic so the Metal build can complete while all other warnings remain fatal. Assisted-by: Codex:gpt-5 [systematic-debugging]
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
The
vllm-cppbackend could configure four of the engine's knobs — block size, KV block count, max sequence length, max concurrent sequences — out of a config surface that is considerably larger. Speculative decoding, prefix caching, the chunked-prefill token budget, the scheduling policy and the external KV connector (LMCache) were reachable from vllm.cpp's own HTTP server and from nothing LocalAI could write in a model config.Part of that gap was the C ABI itself, which carried strictly less than
EngineParamsdoes. That is fixed upstream in vllm.cpp ABI v9 (merged tomainaseec09bed, which this PR pins). The rest was here: the backend parsed a flatoptions:list with five recognised keys and had no way to express a nested JSON document at all.Configuration
Config now goes through
engine_args:— the same map the vLLM and SGLang backends already take — with keys spelled as vLLM's own CLI flags, so aspeculative_configorkv_transfer_configblock written for vLLM works verbatim:The
options:list keeps working and now reads every key too, so no existing config breaks;engine_argswins where both set the same key.All three speculative methods the engine supports are reachable:
mtp(draft head inside the target checkpoint, safetensors only),dflash(separate draft checkpoint,model:required), andngram(draft-free).Two details worth reviewer attention
enable_prefix_caching: falsemaps to the ABI tri-state force-OFF (2), not 0. 0 means "let the model capability decide", and dense architectures default the cache on — collapsing the two would silently enable it against an explicitfalse.cSamplingParamsgrows the ABI v8 logits-processor tail. LocalAI installs no processor, but the C side reads those fields off the pointer we hand it, so a Go struct that stopped atStructuredJSONObject(120 bytes vs C's 136) would have had the engine read past our allocation and call whatever sat there. Latent only because the old v5 ABI gate refused to load a v8 library.Importer
A
vllm-cppimport of a HuggingFace repo now probesconfig.jsonand writesspeculative_config: {method: mtp}into the generatedengine_argswhen the checkpoint declares an MTP head — the safetensors analogue of the llama-cpp importer's GGUF header probe. An explicitspeculative_configis never overwritten, and every probe failure is non-fatal.Two asymmetries versus the llama.cpp hook, both deliberate:
spec_type:draft-mtpwhen the chosen backend isvllm-cpp. Those are llama.cpp option keysvllm-cppdoes not read, and vllm.cpp rejects MTP over a GGUF source outright because themtp.*draft tensors do not survive GGUF conversion. This was a pre-existing bug — a GGUF import with thevllm-cpppreference emitted dead llama.cpp options.Docs
docs/content/features/text-generation.mdgains avllm.cppsection covering theengine_argstable, all three speculative methods, LMCache, and the legacyoptions:list. The backend previously had no documentation page at all.Testing
make lintclean.core/config,core/gallery/importers,backend/go/vllm-cppall green. 9 new importer specs, 11 newengine_argsspecs, plus updated struct-offset assertions for the v9 / v8 layouts.libvllm.soat the pinned commit and confirmed purego binds all 19 symbols withvllm_abi_version()returning 9. That check is now a spec: setVLLM_CPP_LIBRARYand it runs without needing model weights, so a future pin bump with a stale struct mirror fails in CI rather than at a user's first load.Not verified: a real inference end-to-end. The new knobs are checked at the ABI-handshake and unit level, not against a live generation with weights loaded.
🤖 Generated with Claude Code